Seoul , South Korea -LRB- CNN -RRB- -- Four members of a group of 31 North Koreans who accidentally crossed over into South Korean waters on a fishing boat , have decided to defect to the South , despite Pyongyang 's demands that they all be repatriated to the North , the South Korean Red Cross said .

The defection of the four North Koreans could spark tensions on the peninsula , where joint U.S.-South Korea military drills have kicked off this week . North Korea had threatened to engulf Seoul in a `` sea of flames , '' a day before the opening of the exercise .

The South will return the remaining 27 members of the group through Panmunjom , a truce village on the ground border between the two sides , and send the fishing boat back through the waters in the West Sea on Friday , the Red Cross said .

South Korea 's intelligence agency questioned the North Koreans for almost a month since they crossed over in February before deciding to release the group , the aid agency said .

There are currently more than 20,000 North Korean defectors in South Korea , according to the South 's Unification Ministry . The number of annual defectors has risen dramatically since the turn of the century and continues to climb .

Defectors who are forcefully repatriated to the North after attempting to cross over into third countries such as China , Mongolia or Southeast Asia are commonly sent to prison camps as a punishment . Many defectors who live in South Korea fear for the lives of their family or relatives who remain in the North .

@highlight

A boat with a crew of 31 crosses into South Korean waters , the Red Cross says

@highlight

Four of the fishermen decide to stay in the South

@highlight

The number of defections has risen dramatically in the past decade